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The paper addresses the use of heterogeneous ensembles of intelligent unmanned vehicles in such a perspective 
field of innovations as an unmanned fishery. The issues of joint activity of unmanned vehicles of different types in 
fishing operations based on intelligent technologies are investigated. The “smart fishing” approach based on the joint 
fishing operation model is proposed. The operational framework that includes missions, roles, and activity scenarios 
embedded in the discretized spatial model is presented. The scenario activities are considered as the sequences of pen- 
tad that determine executing specific functions concerning the specified waypoint, timepoints, and the states of vehic- 
les. The definition of the plan as the scenario prototype that needs adjusting to the conditions of the situational context 
is proposed. The coordination problem regarding the joint activities of the unmanned vehicles and their scenarios is de- 
fined and the coordination framework based on the distributed common board model and coordination primitives is 
presented. The prototype of the intelligent scenario-based system including the implementation of both operational and 
coordination frameworks developed for the control of unmanned vehicles is described. This system makes unmanned 
vehicles capable to absorb all the latest advances in intelligent technologies to perform smart fishing operations jointly 
in a large heterogeneous group. The proposed approach to smart fishing using intelligent technologies makes it possible 
to detach fishermen from the fishing activities dangerous to their life and health, to reduce significantly poaching and 
illegal fishing, to increase the overall efficiency of fishing operations, and even to save the marine ecosystem. 
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Y cTatTi posrisHyTO MMTaHHA BAKOPHCTaHHA pi3HOpiAHHX aHcaMOsB iHTeIeKTYaIbHHX Oe3iiIOTHHX allapaTiB 
y Takili NepcieKTUBHI rasy3i IHHOBaLii 1K Oe3sniOTHe puOaBcTBO. JJOcIIWKeHO Pi3Hi ACICKTH CHWIbHO! WidIbHOCTI 
Oe3IWIOTHHX allapatTiB pi3HHX THINIB y pHOabCbKUX Olepalli#x Ha OCHOBI iHTeJIeKTYaIbHUX TeXHOJOri. 3amponouo- 
BaHO TLAXi, 0 “pO3yMHOLo pHOaIIbCTBAa»), 3ACHOBAHUM Ha MOJeIII CIIbHOI puOaIbcbKOi Olepallii, IpescTaBJIeHO OTle- 
paliitHuit dpelimBopk, AKU BKOYAE Micii, pos Ta ClleHapii iMbHOCTI, BOyZOBaHi B JMCKpeTH30BaHy IIpOCTOpoBy 
Mojesb. JIisIbHICTh 3a CICHAPIeM POSPIATA€TbCA AK MOCIIAOBHICTh MeHTAa, AKI BU3HAYaIOTh BUKOHAHHA KOHKPeTHHX 
(PYHKUWI Y KOHTCKCTI 3a3HaYeCHHX TOOK WWIAXy, YaCOBHX TOOK Ta CTaHy Oe3MiIOTHUX allapaTiB. 3allpomoHOBaHo BH- 
3HaYeHHA IlaHy AK IPOTOTUMy ClleHapiio, AKUM WOoTpeOye IpHcTOcyBaHHA JO YMOB, BH3HaYCHHX KOHTCKCTOM CHTya- 
wii. BusHayeHo mpoOemy KoopyMHali CIiIbHOI AKTMBHOCTI Oe3MJIOTHHX allapaTiB Ta ix cl{eHapiiB, a TaKOxK Mmpey- 
CTaBJIeHO KOOpAMHAMIMHHM PpeMBopK, 3aCHOBaHHi Ha pO3MOAWIeHi MOE CHIbHOI JOWKM Ta IPHMiITHBax KOOp- 
uHawii. OnMcaHo NpOTOTHM iHTeseKTyabHOi CHCTeMH, WO Oa3yeTbCA Ha Cl{eHapil, AKUM BKIOUAE peai3zalllro Orepa- 
IMHO Ta KOOpAMHaliHoro :PpeiMBopKiB Ta po3spoOJeHHH JIA yIpaBIiHHA Oe3sMiIOTHUMH allapatamu. La cuctema 
poOnTs OesIiJIOTHi allapaTH 34aTHHMU yBiOpaTH B ceOe BCi HOBITHI JOCATHCHHA iHTeIeKTYaIbHUX TeXHOJOTIM [A 
CIIbHOTO 3iiicHeHHA «“pO3yMHUX>> PHOOJIOBECLEKHX Ollepallii y BeJIMKHX HEOAHOPIAHUX rpynax. 3anponoHoBaHHit 
TAXI JO “po3yMHOro pvOaIbcTBa» 3 BAKOPHCTaHHAM IHTeJICKTYaIbHUX TEXHOJOFIM Jae MOXMBICTS BICTOPOHHTH 
puHOanoKk Bid HeOe3Me4HHX Id iXHbOrO XKUTTA Ta 340poB'A Oepallilt pHOOMOBI, 3HaYHO 3MEHIIMTH OpakOHbEpcTBO Ta 
He3aKOHHY PHOOJOBIIO, MABHUIMTH 3arasibHy e(PeKTHBHICTh PHOaIbCbKUX OMepalili i HaBITb AKHaliKpalje 30eperTu 
MOPCbKy eKOCHCTeMy. 

Ksn040Bi c10Ba: orepallia “po3yMHOro pHOaIBCcTBa», Oe3MWIOTHHM allapar, MICiA, POJIb, CleHapili, KOopAMHALiA, 
CHHXpoHi3allla 
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Introduction 

Fish is an important food for all man- 
kind. Thus, the fishery is one of the most im- 
portant food industries. Due to its growing de- 
mand, the fishing fleet increases from year to 
year, and the total fish catch has also grown. 
Today, we observe a global trend toward in- 
creasing fish consumption. However, the 
growth of industrial fish catches and their pro- 
cessing is limited both by the recent decrease 
in fish livestock and several climatic, environ- 
mental, and technological factors. Therefore, 
last year's fishing volumes are large enough 
but variable. An increase in the volume of fish 
catches does not depend further on the number 
of fish fleets or the intensity of its use. Now 
the volume of the fish catch is regulated by 
quotas due to the requirements of saving the 
fish population. Accordingly, the efficiency of 
the fishing industry that has been traditionally 
estimated by the catch per fisherman is drop- 
ped in recent years, and this is essentially the 
result of the lack of development in fishing 
gear and technologies [1]. It is clear that the 
fishing industry needs modern fishing gear 
and technologies. Unfortunately, this industry 
has a relatively low level of innovation due to 
objective reasons. Most of the modern sophis- 
ticated technologies are not mentioned in ma- 
ritime international codes and conventions 
(i.e., SOLAS, UNCLOS, COLREGS, STCW, 
ISM, IMO, etc.) and this hinders the use of un- 
manned vehicles equipped with onboard sen- 
sors and gears, which can significantly im- 
prove the efficiency of the fishery. Besides, 
the active use of unmanned technology will al- 
low a significant number of fishermen to re- 
main healthy and even to save their lives since 
their work is a source of increased danger. 
Nevertheless, the systematic renovation of ma- 
ritime international codes and conventions is 
been conducted, and one can hope that the op- 
portunities for innovation in the fishery will 
significantly increase soon. 

Problem statement 

The future of fisheries does not depend 
on any individual technological innovation, 
but modern technologies must complement 
and communicate with each other to help in 
effective fishing. There are plenty of recent 
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fields of innovations, the results of research 
and development in which can be exploited in 
fishery, such as Artificial intelligence, Ma- 
chine learning, Machine vision, Remote Sen- 
sing, Big data analysis and data processing, 
Unmanned vehicles, etc. 

Modern fishing operations are based, 
first of all, on the search for fish schools. Next, 
it is necessary to identify fish species and sizes 
of fish found in the fish school, because a 
school with main juveniles should not be 
caught, it should remain in the ecosystem for 
the fish reproduction. Non-commercial fish 
varieties should also not be thoughtlessly 
caught. If a school of fish is determined to be 
suitable for fishing, then the fishing boat must 
be navigationally focused on the effective use 
of special fishing gear to optimal catching of 
fish. Of course, this is actually a very simpli- 
fied scheme of the operation, but it is clear that 
unmanned vehicles (UVs) can be the most ap- 
propriate means to accomplish such an 
operation. 

It seems that the use of the unmanned 
vehicle in an individual manner cannot play a 
decisive role in providing a technologically in- 
novative and efficient fishing operation. 
Thanks to continuous improvement, UVs have 
become much more autonomous and intelli- 
gent than just a decade ago. It is also important 
that they have become much cheaper, which 
ensures the mass use of UVs in large groups. 

Therefore, the problem addressed in this 
paper relates to the use of large groups of mo- 
dern intelligent UVs capable of absorbing all 
the latest advances in the above-mentioned 
fields of innovations. We will investigate the 
issues of joint use of UVs of different types in 
innovative unmanned fishery operations. 

Related works 

Unfortunately, there is very little litera- 
ture on the use of UVs in fishing. Most resear- 
chers have investigated fishing issues that are 
loosely related to this topic. For example, 
there are many works devoted to managing the 
sustainability of fishery resources [2,3] or 
smart fish farming [4]. Nonetheless, there are 
good reviews on innovative technologies in in- 
dustrial fisheries [5, 6], but their consideration 
is restricted to the use of UVs for monitoring 
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and patrolling fishing areas, so they weakly 
address the topic. Several works have also 
been devoted to the investigation of intelligent 
tools and technologies for monitoring, control, 
and surveillance of unwanted catches [7, 8] as 
well as surveillance task for searching and 
counting fish in fisheries’ video [9, 10]. There 
are also a few papers that address the develop- 
ment of smart fishing gears based on traditi- 
onal fishing gears such as trails, seines, nets to 
catch fish as well as acoustic sonar and echo 
sounders to detect fish [11-13]. Furthermore, 
there are also exotic attempts to fish directly 
from aerial UVs [14], but this is more of a 
sports and recreational activity, rather than in- 
dustrial fishing. 

The automated fishing system has been 
proposed in [15] that offers an effective me- 
thod for harvesting the coastal pelagic fish re- 
source. Indeed, it would be the first unmanned 
fishing system, but it did not use UVs because 
it has been coastal. Some interesting ideas of 
using underwater UVs equipped with side so- 
nar, multibeam sonar, and environmental sen- 
sors (oxygen, temperature, salinity, turbidity, 
camera) for fishing and fish farming have 
been proposed in [16]. This paper listed requi- 
rements for onboard sensors and emerging 
communication (navigation) problems for 
underwater UVs. However, it considers only a 
task of trawl net following related to the 
fishing operations. 

The ultimate smart boat vision has been 
proposed in [17] based on rapid change in 
such technology as networks, sensors, and 
artificial intelligence. 

New ideas related to the implementation 
of innovative intellectual methods in fishing 
operations lead to interesting projects such as 
SmartFish [18], which is aimed mainly at im- 
proving catch efficiency, catch composition 
and catch quality in pelagic and demersal 
fisheries, reduction in fishing mortality, the 
capture of protected species, and the environ- 
mental impact of fisheries. Within this project, 
plenty of smart technologies have been pro- 
posed dedicated to pre-catch size and species 
recognition based on optical and _ hydro- 
acoustic technologies (SeinePrecog), real-time 
detection of organisms that are undetectable 


using conventional fish finding techniques 
(i.e., echo sounders and sonars) (FishFinder), 
providing detailed information on species and 
sizes entering the trawl (TrawlMonitor), using 
LED technology to optimize the catching per- 
formance of trawl fishing gear (SmartGear), as 
well as a hardware and software infrastructure 
for acquisition, analysis, and presentation of 
data from onboard catch monitoring systems 
(FishData). 

It is known that UVs and sensors are be- 
coming smaller, cheaper, and more powerful, 
so it is now realistic to combine all modern 
technologies centered around UVs. Since UVs 
find more and more applications and cover 
more and more fields in the sea and ocean en- 
vironment, it's time to consider their applica- 
tion in such a field as smart fishing. 

The aims of the paper 

Considering the above-mentioned rea- 
sons, the most topical issue for today is an in- 
vestigation of intelligent technologies in the 
context of smart fishing operations. We sup- 
pose that smart fishing operation is performed 
by a large group of UV. Such a group can in- 
clude underwater, surface, and aerial UVs si- 
multaneously, so it can be heterogeneous. 
Since each of the UV plays its role within the 
group, performing a certain individual mission 
during a common joint fishing operation, such 
structure of a group of UVs is called an en- 
semble, which distinguishes from other cur- 
rently known structures of UV groups (Le. 
swarms, flocks, etc.) [19]. Thus, the paper 
aims to propose an approach to smart fishing 
using the heterogeneous ensemble of UVs 
with proper coordination. 

Methodology 

Let us imagine an intelligent fishing 
operation (smart fishing) performed jointly by 
a wide range of UVs of various types and clas- 
ses involved in the implementation of certain 
missions based on their competence. For 
example, unmanned aerial vehicles can be in- 
volved either individually or in small groups 
in the search for schools of fish. Unmanned 
underwater vehicles, either individually or in 
groups, can effectively use sensors to identify 
species of fish and their numbers in schools, 
assessing the feasibility of catching them. Un- 
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manned underwater vehicles have other group 
tasks such as driving fish schools to fishing 
gear, trawl net following in order to estimate 
the volume of the catch and the occupancy of 
the trawl net. Unmanned surface vehicles (i.e., 
unmanned boats) can carry fishing gear, so 
they also can be used individually, in pairs, or 
small groups depending on the kind of gear. 
Naturally, we need to use a relatively large 
vessel in the fishing operation as a carrier of 
all the above-mentioned UVs that allows them 
to port (take off or land), recharge, unload the 
catch, etc. Such a vessel can accumulate and 
process the caught fish, and of cause, it can be 
serviced by a team or can also be unmanned 
soon. 

Obviously, the overall goal of such 
fishing operation will be to catch selectively a 
given volume of fish of certain species and 
sizes, with a minimum expense of time, fuel, 
oil, etc., and, of course, with a minimum nega- 
tive impact on the ocean environment. In this 
context, UVs jointly, sequentially, and synch- 
ronously perform assigned individual or group 
missions until the overall goal of the operation 
is achieved. The assignment of missions to un- 
manned aerial vehicles is associated with their 
functional and technical capabilities, which 
determines their ability to perform certain ro- 
les. UVs can be universal, allowing them to 
perform several different roles, or specialized, 
adapted to a specific role. Thus, let us define 
the joint fishing operation model and the hete- 
rogeneous ensemble of UVs concerning its 
composition and structure. 

The joint operation models 

Suppose U is a set of UVs, F is a set 
of their parameters, ® is a set of their func- 
tions, and Cl is a set of classes of UV. Each 
UV u,€U belongs to a certain class cl, € Cl 
according to its parameters and capabilities 
and can perform a certain set of functions 


D, ={Pox+Pu>-Pne }- This means that u, can 
perform at least one function g, <¢® depen- 


ding on its class. 
Let T be a timescale constructed over 
an open ordered set of time points [aeeiel al 


Suppose G is a set of groups of UV. Each 
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group g,¢G, in turn, is a dynamic set of 


UVs, since at different time points t<e7 such 
a group can consist of different numbers of 
different UVs. The state of each UV u, EU 


can be described by a dynamic set of parame- 
ters’ values (i.e., speed, direction, coordinates, 


etc.) F(t)={f,(t),--fe(0)} 
Let = be a three-dimensional Euclidean 


space discretized by a metric grid D of coor- 
dinate lines with size 6, certain metric €, 


and linear mapping f such that coordinate li- 


nes constitute a set of cells having the size 
0xdx6, f:=—D. As the result, we obtain 


the grid D ={d..} of isometric cubic cells 
d 
In this way, each cell d,,. €D is a spatial ob- 


where x,y,z meet the cell coordinates. 


xyz ? 


ject of the smallest size. Therefore, the dis- 
crete location of each UV, obstacle, or goal is 
geo-referenced to the certain cell(s) within 
space =. Cell size 6 can usually be deter- 
mined by the technical capabilities of UVs’ 
sensors and the computing capabilities of their 
onboard computers. 


Suppose each UV u, €U can perform a 
motion function g, €® moving inside space 


al 


=, changing its state F(t) over time, and 
avoiding both obstacles and collisions with 
other vehicles. It is clear that UV u,<€U can 


also perform any other function 9, «® 


depending on its class. 
Let Pos(u,) be a position of UV u, €U 


within a discretized three-dimensional space 
D, so (Pos(u,).t,)=(dy.), dy. €D. Let TP 


be at certain points in time, WP be a certain 
waypoint, and FP be a certain state of the ve- 
hicle. Clearly, the route of each vehicle can be 
represented as a certain sequence of waypoints 
WP. with respect to the corresponding 7P. In 
other words, the route can be defined as an or- 
dered array of pairs (TP,WP) as a convenient 


way to control and coordinate the joint activity 

of the vehicles, primarily their joint motion. 
Assume that any WP can also be repre- 

sented with a certain approximation as a cell 


74 © B.T. Wepcriox, M.B. Kapixosa, I.B. Coxon, P.M. JleskiscpKuii, B.M. Tyces, 1.0. JopospcpKka 


ISSN 1561-5359. Wtyannii intenext, 2020, Ne 2 


d,,, €D . Thus, (1,,Pos(u;))=(TP,WP),,,. and 
a trajectory of UV u, during the time interval 
te[t,.t,,] can be determined as a sequence 


| (TP.WP),,.-{TP.WP), | 


Suppose that UV u, «U must take a cer- 
tain position Pos(u,) at the time point ¢, ¢T 
to perform a specific function g, ¢® (it will 


be convenient to distinguish functions that can 
be performed only once in a particular WP 
(e.g., unload the catch) from functions that can 
be performed continuously during the motion 
of the vehicle from one WP to another (e.g., 


trailing). 
Accordingly, a tuple (pentad) 
(1, Pos (u,)Fi(th)-@ it) (1) 


determines that UV u, ¢«U has to perform a 
specific function g,<¢® over time point 
t,; €T previously taken the position Pos(u;) 
(WP ) and having the state F,(t,) (FP) at the 
time point 1, ¢T (TP). 
Clearly, a sequence 
“4 (1,,Pos(u,),F (ujst;)Qjot) )-- 
Tr (u, i = 

(t,,,Pos(U;).F (Ujsty )sPpote ) 
is called the trajectory of the vehicle’s u, acti- 
vity at a time interval re[t,,t,,] that deter- 


mines completely the activity of the specified 
UV at a given time interval. It can also be re- 
presented as a vector 


((WP,TP, FP), syst; )~ 
Tr(u, = 
((WP,TP, FP), »Peste) 
Consider a multitude of groups 
815-8, €G and separate UVs w,..u, €U 
dispersed over the space = perform there a 
certain joint operation Op aimed at catching 
fish from found schools. Let H be a set of 
fish schools. Thus, the position of each foun- 


ded fish school h, ¢ H over time can also be 
represented approximately within the certain 
cell d,,. € D by the pair (1,,Pos(h, )). 
Suppose long-term operation Op con- 
sists of a set of missions M ={m,..m,}, 


which can be performed either sequentially or 
in parallel. Each mission m,¢M_ can be as- 


signed to a certain UV u,€U or a certain 
small group g,<¢G. Let R be a set of roles. 
The mission m,€M can assume one or more 


specific role(s) that should be assigned to se- 
veral UVs performing them jointly. Thus, the 
set of assigned roles R must cover the entire 
set of missions M ={m,,...m,\ required to 
operate Op. 

Consider the UVs U,, = {u,,...u,} CU of 
corresponding classes {cl,,...cl,}<Cl united 
in some groups G,, ={8,,--.8,} CG. If UVs 
{u;,..u,}eU,, perform certain missions 
M,, ={m,,..m,}<M with assigned roles 
R,, ={hi--l,}eR in the context of a certain 
operation Op, we assume that UVs 
{u,,..u,eU,, Constitute a heterogeneous en- 


semble En defined by the sets of participating 
vehicles U,, and groups G,, as well as the 


sets of operational missions M,, and roles 
R,,- Clearly, the ensemble En must have a 
tree-like structure Str(En) and shape 
Shp(En); the last determines its dynamic 
spatial configuration V(En,t). The structure 
Str(En) represents a_ structural aspect, 
whereas the shape Shp(En) represents a geo- 
metric aspect of the ensemble concerning the 
performing operation Op. Thus, 
En= (OP.U 4y+Gyy»M 4 -R,,»Str (En), Shp(En)) . 

If the certain UV u, €U,, is a member 
of the En, it must perform a certain sequence 
of assigned missions M, =[m,,,...m,,] accor- 
ding to the roles R, =[T,.,---7, ]- 

Each role 7,¢R within the mission 
m,,€M_ can be implemented through the exe- 
cution of a certain activity scenario Q,,. 


Usually, the scenario requires that UV must 
move according to a certain sequence of posi- 


tions [ WP, WP,, | and perform the specified 


12% 


actions in the given WPs/TPs. In this case, 
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the trajectory of the activity Tr(u,) is the 
objectification of the scenario Q,, during its 
execution time. 

Thus, the scenario of the certain UV 
mission can be represented as the desired tra- 
jectory of its activity determined by a se- 
quence of waypoints associated with the cor- 
responding time points, states (represented as 
a set of parameters such that speed, direction, 
etc.), and actions (related to the performing of 
the specific function(s)). 

According to the scenario, UV u, €U,, 
at each defined TP must be located at the cor- 


onl 
=) 


responding cell within space = (WP) having 
specified state (FP) (optionally) and perfor- 
ming the specific function 9, <€@ or the set 


of functions {¢,,5...9,} €® . 

Let us denote the certain pentad (1) by 
p, such that (t,,Pos(u,).F,(t,).~;-t;). The 
scenario Q,, can be represented as a sequence 


of pentads Q,, =[,;,---Py |, where 1; <t,. 


Mission 
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The operational framework 

Consider now the operation as a whole. 
The scheme of the fishing operation 
Sch(Op)={m,,..m,} contains a set of mis- 
sions that should be assigned for the certain 
UVs {u,,...u,}€U,,, Which are the members 
of the ensemble En concerning the operation 
Op. Some of the missions can be performed 
in parallel but others in sequence, after com- 
pleting another mission. In its turn, each mis- 
sion m,<¢M,, contains a set of roles 
{Tis } Em, - A certain UV can be involved 
in the implementation of the specific role 
based on the relevance of its functional and 
technical capabilities. In its turn, each i-th 
role r,, within a certain mission m, can be re- 
presented as a multiplicity of alternative acti- 
vity scenarios 1, = (Oz tal Qin) . Such a tree- 
like scheme of the operation is named an ope- 
rational framework (Fig. 1). 


Mission 
n 


Fig. 1. The operational framework structures 
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It can be also represented as 


{Coord beg ae rere (oe (ae | ee 
Oa ls coe pe (Q gale ucoeee |, 


where the first index of each scenario repre- 
sents the mission within the operation Op , the 
second index represents the role within this 
mission, and the third index represents the 
alternative of the scenario within this role. 

The multitude of all scenarios executed 
jointly and simultaneously within the operatio- 
nal framework distributes the activity of the 
ensemble En in space and time. Thus, the re- 
ciprocal location of WP s of all involved UVs 
{u,,..u,}€U,, at the certain TPs that corres- 


Sch(Op) = 


pond to the specific time point +, «7 determi- 
nes the spatial configuration V(En,t,) of the 


ensemble En. 

The structure Str(En) of the ensemble 
En depends on the operational framework 
Sch(Op) while the latter depends on the 
change of spatial positions of all members of 
the ensemble En and their roles 7, ¢R,, as 
well as on the change of locations of the 
founded schools of fish 
Pos(h,.t,). h, €H, t,€T. Further, the spatial 


positions of all members of the ensemble En 
constitute its scheme Shp(En). This scheme 
depends significantly on the current situation 
Sit , which, in turn, is determined by a combi- 
nation of the spatial distribution of the 
founded schools of fish, the spatial configure- 
tion V(En,t) of the ensemble, states of UVs, 
state of the environment, given constraints, 
etc. 

Let us define a __ procedure 
S(U or); that assigns the given role 


4; €R,, to the most appropriate UV u, €U,, 


based on the relevance of its functional and 
technical capabilities and a_ procedure 


Ss(tiyo%) OQ, that choose the appropriate 


scenario ©,, based on the given role 7, €R 


op 


and its performer u, €U,,, 


Suppose each UV u, €En has a rele- 
vant pre-built plan P/(M,) for performing 
each feasible mission m,¢M,,, represented as 


a certain prototype of scenarios in the context 
of the current situation Sit. In this way, the 
fishing operation Op can be imagined as the 
implementation of a certain mutual plan 
PI(Op) = Pl(m,)o...c Pl(m,) consisting of the 


plans for individual missions m,,...m,<¢M 


op 

executed by UVs u,,...u,¢€U,, according to 

their roles r, € R,, within the ensemble En. 
Each prototype P/(m,) is the initial 


plan that can be determined by the appropriate 
scenario ©,, indicating the sequence of re- 


quired positions and functions performed but 
having only abstract (formal) parameter va- 
lues. As soon as the actual parameters will be 
substituted in place of formal parameters cor- 
responding to the current situation Sit, this 
plan will be implemented to a specific scena- 
rio of activity containing the trajectory of acti- 
vity Tr(u,). Thus, the initial plan of the entire 
fishing operation P/(Op) , as well as the rele- 
vant mission plans P/(m,) of the individual 
UVs u, <¢ En, should be defined based on the 
mutual spatial position of the vehicles (perfor- 


mers) and positions of the detected schools of 
fish h, that can be targeted for these missions. 


The implementation of the above-men- 
tioned plans can usually be disrupted. Clearly, 
the change of the positions of any moving ob- 
jects, ie. vehicles, obstacles, and fish schools 
lead to the change of the plans by updating the 
values of the actual parameters, which in turn 
leads to reprogramming or even changing sce- 
narios. Furthermore, due to the unpredict- 
ability of the environment (weather condi- 
tions, unforeseen obstacles, etc.) UVs u, € En 
are exposed to many dynamic and situational 
disturbances, so their planned trajectories 
Tr(u,) can be violated. They also must avoid 
collisions and obstacles by maneuvering; this 
also requires the proper correction of the as- 
signed scenario Q,,. Thus, the plan P/(m, ) 
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often needs to be changed "on the fly" by re- 
placing, adding, or removing some pentads 
P, Within Tr(u,). 

The planned trajectories of the UVs u, 
and u, to perform the mission of the catch 
school of fish h, are shown in Fig. 2, a. 


a) uo 


ae 


WPo2 = WPi4 
y a , - — WPi3 
¢ e 
“s yi ¢ / WPi2 
WPo1 6 WPH 
b) uo 
WPos A 
ve es A 
04 e A I 
S, 3 co ul 
a” d 
WPo 9 “2 ie 
Pug 6 h’) : WPi; 
WPo2 a” re Zt 
=- . 
Z 7 WP. 13 = - Yy 
r 7 4 WPi2 WE 
yY 
WPo1 6 WP 


Fig. 2. Changing the scenarios of the UVs 
under the current situation 


The transformation of the trajectories of 
both UV u, and u, in response to the situatio- 
nal perturbation caused by the change of the 
motion direction and speed of the school of 
fish h/ is shown in Fig. 2,b. The response to 
the situation that occurred is obtained through 
the changes in the waypoints 
WP,,, WP,;, WP.,, WP, corresponding to the 
time points 7P,, TP, . 

Clearly, there is always a set of restric- 
tions L={i,,../,} on the distances, bearings, 


angles, relative speed, and other parameters 
imposed on the UVs’ joint activity due to the 
used fishing gear, marine rules, and regula- 
tions, etc. (Fig. 3). Therefore, any changes in 
scenarios or trajectories of the UV should not 
violate the specified restrictions L. 
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WPo1 


Fig. 3. The trajectories of the vehicles u,,u, 
with restrictions 


Suppose the scenario is a changeable se- 
quence of actions Q, =[dy,,--.d,, |, where 


each action a,, can be considered as a certain 


change in the values of UV’s activity para- 
meters (e.g., motion speed or direction). Let 
any change of any parameters of the joint acti- 
vity of UVs within the ensemble En be consi- 
dered as events. 

Every event that arises within = cause 
the situation Sit. Since the conditions for the 
execution of the current scenario Q,, for the 


certain UV u, € En can be broken, definitely, 
there is a need to respond to the event in a 
timely and adequate manner. 

Such response to the event can be im- 
plemented by changing the mission plan 
Pi(m,), rebuilding, adjustment, adaptation of 


the current scenario Q,, through addition, re- 


placement, or removal of certain pentads, sear- 
ching for a new appropriate scenario Q),, or 


even changing the shape Shp(En) or the 
structure Str(En). 


Since the arising event can lead to a 
change in the conditions of the scenario exe- 
cution not only for individual UV but for seve- 
ral UVs or even an entire group of UV at 
once, all of them can begin to simultaneously 
react to the event by changing their plans and 
scenarios. Thus, the main challenge of the 
fishing operation Op is the joint activity of the 
performers {u,,...u,,}¢U,, embodied in the si- 


multaneous change of their scenarios ,,...0,,. 
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Since the changes in the scenario for one of 
the UVs lead to a change in the conditions for 
the scenario execution for the other UVs, there 
is a need to solve the coordination problem for 
their activity trajectories Tr(u,),...7r(u,) that 


should be synchronized in time and space. 

Coordination problem 

Let us define a coordination problem. 
Consider the set of timepoints TP cT. Sup- 
pose <, is a strict order and &, is a metric im- 
posed over T. 

Suppose the set WP is a set of way- 
points divided into two disjoint subsets: the 
subset WP, of waypoints that are capable of re- 
location and the subset WP, of waypoints, 
which can not be relocated. The set WP, is 
always related to either target (final points) for 
UVs’ motion or waypoints, within which UV 
should perform any function @, that differs 
from @, . In the latter case, if UV needs to per- 
form a continuous function @,, its activity tra- 
jectory must contain both initial and final way- 
points that can not be relocated. The other 
waypoints can be considered as capable of 
relocation. 

An example in Fig. 4 shows the activity 
trajectories of two vehicles, uy and u,. The 
waypoints capable of relocation are shaded 
there while hollowed waypoints can not be re- 
located. The necessity of relocation of WPs 
arises in response to the events. Since events 
caused by certain disturbances appear (or dis- 
appear) dynamically, the coordination task 
arises (or terminates) sequentially. 


WP.2@ uz @WP34 
WP3;% 


Fig. 4. Adjusting the scenario by the 
relocation of the waypoints for UV u, 


The coordination task J, for UV 
u, € En at the given TP t, can be solved by 
the procedure that adjusts the executing scena- 
rio Q,, corresponding to the situation Sit at the 
moment t,. The solution obtained by this pro- 


cedure is a time-ordered sequence of actions 
esas) (aes bin eg ey )| within the sce- 
nario Q,,. Obviously, all WP's within Q,, cor- 
responding to 7Ps t,,t 


ptjaetj,, Will be relo- 
cated in the context of the performed actions 
G;,,4;>,-..d,,. Moreover, WP s related to the sce- 
narios of the other UVs of the ensemble En 
might also need to be relocated. 

To obtain such a solution, we should 
first build a network using a partial order rela- 


tion <, imposed over the set of coordination 
tasks such that 3=({I,,,/,,.--l,,}.<,)~ The 
constraints related to the nodes of this network 
should be tested on their satisfaction looking 
through 3 in ~<,-ascending order. If some 
constraints are violated, the coordination task 
must be solved for the corresponding pair of 
vehicles. We assume that each 3 node con- 
tains a set of constraints {/,,,../,,t¢L imposed 
on the activity of UV u,, and a procedure ¢,, 
can be used for checking the constraint /;, sa- 
tisfaction. Thus, the set of synchronization 
points (SP) should be determined and adjusted 
in time and space to solve the coordination 
task. 

The strict order <, allows us to define 
for each u, and wu, such synchronization 
points t, that 34,,t;|t, =ts A t; =ts. Since 
one can look through the scenarios Q,;,Q,, in 


<,-ascending order, all 7Ps t,,t 


poe jab 


ti, can 
be enumerated to use them as synchronization 
points. The set of the possible WPs 
Ww = {Wyse My, | EW, for UV u, can be ob- 
tained based on the cinematic models of UV 
u, and the corresponding set of WP's w, €W, 


obtained based on the cinematic model of UV 
u, (Fig. 5). One of them should be chosen to 
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relocate corresponding WP s within the activity 
trajectory Tr(u,) in such a way that the set of 


imposed restrictions L must not be violated. A 
good idea is to use the constraint-satisfaction 


events ees 


I 
—>(ui)—> situation 
\ 


=——_ = 


trajectory 


ux; cinematic 
model 


@) u, <, Uy; 


based control 


trajectory 
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approach considering synchronization points as 
variables and w’ as its domain we use. 


scenarlo- or ae a 


scenario 1 


system -4-7 


: : I 
ujcinematic |, _. _ 
model 


TP in WP im 


constraint 
satisfaction 
procedure 


Fig. 5. Process of pairwise vehicles coordination 


The solution of the coordination prob- 
lem K :({Q,,...Q,,}) > {Q/,..Q/,} for the en- 
tire set of UVs {u,,...u,}eU 


op Participated in 
the ensemble En is to arrange the necessary 
synchronization points within the correspond- 


ding scenarios 9,,,..Q,,, using the constraint 


satisfaction method assuming that SPs are va- 
riables and WPs are their domains for each 
pair of the vehicles u,;,u, En, which satisfies 


the set of given constraints at each TP such 
that 


((WP,TP) J. (WPLTP) 5) 


“ene 


((WP.TP), ,)-(WP.TP),. 


(i 


Clearly, the accuracy of the solutions of the 
coordination task essentially depends on the 
accuracy of the initial information. 

The coordination framework 

Regardless of the moving environment 
UVs participating in the fishing operation are 
equipped with a set of sensors to observe the 
operational space and monitor the activity of all 
objects within this space. For example, aerial 
UVs use high-resolution cameras to provide 
surveillance and avoid collisions, but under- 
water UVs use side and multibeam sonars, oxy- 
gen and temperature sensors, cameras, etc. 

UVs also need navigation to accurate 
geo-referencing of vehicles, obstacles, and 
schools of fish. Aerial UVs use GPS receivers 
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and inertial measurement units for self-locali- 
zation and navigation while underwater UVs 
use compasses, doppler logs, ultra-short base- 
lines, etc. 

Regardless of the type of sensor, it has 
limited accuracy, and the number of sensors in 
UV is also limited for technical reasons. This 
causes only partial visibility of the environ- 
ment that leads to inaccuracy, incompleteness, 
and uncertainty of observations. 

However, the overall efficiency of the 
fishing operation depends on the success of 
finding relevant activity scenarios for all mis- 
sions’ performers, that definitely depends on 
the completeness and accuracy of available in- 
formation to seek appropriate scenarios. In 
other words, given the limited observations 
captured by sensors, the coordination of UVs 
activity relies not only on the synchronization 
of their activity scenarios provided through 
adjusting their synchronization points but also 
on the synchronization of information about 
the operational space observed by individual 
UVs within the ensemble. Thus, UVs need to 
know more about the positions, states, and ac- 
tions of other UVs within the common opera- 
tional space, both for safety reasons and for 


e 


e 


attributes 


e 


the reasons of successful completion of their 
missions. 

Consider the above-mentioned three- 
dimensional Euclidean space = and its discre- 
tized spatial model D. 

Let us define a model of the distributed 
common space of UVs observations in the 
form of a common distributed board V . Each 
item y on board ’ corresponds to a specific 


cell d,,. <¢D. Thus, each element denoted by 


coordinates (x,y,z) is a structure 


Woe = {Xx peak aa , which contains the values 
of the attributes that determine the observed 
state of the cell d,,. (Fig. 6). These attributes 


depend on the joint mission of the UVs and the 
process of their activity. Thus, for smart fishing 
tasks the common board can include such attri- 
butes as x, — the species of fish in the school, 


x, — the average weight of fish in the school, 
x, — the current position of the fishing gear 
(identifier), x, — the current position of UV 
(identifier), x, — the presence of an obstacle, 


etc. Each attribute can be expanded into a set of 
sub-attributes, for example, x,, — the motion 


speed, x, — the motion direction, etc. 


subattributes 


x1 — fish species 

x2 — average weight of fish 
x3 — fishing gear (ID) 

x4 — UV position (ID) 

x5 — obstacle 


x41 — speed 
x42 — direction 


eee 


Fig. 6. Distributed common board model 


© B.I. Wepceriox, M.B. 2Kapixosa, I.B. Coko, P.M. Jleskiscbkuiit, B.M. Tyces, 1.0. Joposcpka 81 


The distributed three-dimensional com- 
mon board Y provides transparency of infor- 
mation and creates a common information en- 
vironment, which is provided for all UVs that 
are the members of the ensemble En and lo- 
cated within the space of joint operation Op. 


Given the inaccuracy of the sensors, each attri- 
bute of the cell d,,. can be supplemented by a 


certain value w having the range [0,1] to pro- 


vide the confidence of the corresponding fea- 
ture within the cell d,,. according to obtained 


observations. Taking into account that differ- 
rent UVs of the ensemble En can simultane- 
ously observe the operational space = storing 
the values of attributes and parameters in the 
same cell, it is expedient that each parameter 


xX should be stored as a_ tuple 


X,, = (asi) es Oe er ee) where x, 
is a value of the parameter X,, observed by 
i-th UV and yp, is the confidence assessment of 


i-th UV concerning the value of the 
parameter x, . 


Thus, UVs should be able to exchange 
information about the observed attributes of 
the cells through the common board ‘Y . The 
main condition for the proper use of such 
coordination tools as the common board ‘Y is 
the coincidence of the starting point of the spa- 
tial models and cell sizes within the control 
systems of each member of the ensemble. It 
allows exchanging information, transparently 
storing it on the common board. The coinci- 
dence of these parameters can be achieved by 
synchronization. 

Since a distributed approach is used to 
coordinate the UVs, the common board 
might not be a physical object within operatio- 
nal space =, but a certain virtual entity, so 
each UV can have its local image of the vir- 
tual common board. Thus, UVs must keep a 
valid copy of the common board and ex- 
change information with other UVs. Coordi- 
nation primitives (messages) presented in 
Table | are intended for transmission and re- 
ceiving of the observations (e.g., the spatial 
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positions and expected characteristics of the 
found schools of fish, the spatial positions and 
motion parameters of UVs, etc.) through the 
UV’s onboard communication equipment. 


Table 1. Coordination primitives 


Primitives | Function Description 
WIT "Who is UV requests the 
there” recall of other UV 
WIHd., "Who's UV asks who is in the 
i here" cell (i,j,k) 
UV reports that it’s 
IMH d.. wy " 
Wa Dene | thenthe cell Go 
"Starting | UV reports that the 


ORG xyz _ | point starting point has 


here" coordinates (x,y,z) 
"Get the UV requests the 
UP starting coordinates of the 
point" starting point 
UV establishes the 
STG dy "Set the presence of a school 
pack" of fish in the cell 
(i,k) 
UV establishes the 
OTGd,, "Set an presence of an 
obstacle" | obstacle in the cell 
(i, j,k) 
"Assign a 
auee UV sets the target 
ATG d,, 
ue ete point in the cell (i,j,k) 
UV lists all the 
objects observed by 
ISE "I see" him and their 
attributes in the space 
of interaction 
Implementation 


For each UV u, € En, the search for a 
new appropriate scenario Q), under conditions 
determined by the spatial configuration 
V(En,t,) and imposed constraints L is the 
task of the intelligent scenario-based control 
system (ISCS). 

At the input, such a control system ac- 
cepts the sequence of events, which determi- 
nes the current situation (Fig. 7). 
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Fig. 7. Information processing within the intelligent scenario-based control system 


Based on the mission plan P/(m,) and 
the current spatial configuration V(r), the 
ISCS of the UV u, performs the search of the 


appropriate scenario according to the assigned 
role 7, with the procedure ¢,. The search 


result is a scenario Q,, that must be adapted 


within a kinematic model. The separate kine- 
matic model-based unit should be used to im- 
plement such adaptation resulted in the adjust- 
ted activity trajectory Tr(u,), which must be 


further verified to test compliance with the 
specified set of restrictions L. To do such 
verification, yet another separate unit should 
be used to solve the constraint satisfaction 
problem. Thus, if the activity trajectory 
Tr(u,) meets all constraints, this trajectory 
must be converted to a sequence of pentads 
and transmitted to the low level of the UV 
control system. Otherwise, if some constraints 
are violated, the ISCS should try to adapt the 
scenario Q,, to the new conditions defined by 


Sit using a small feedback loop (Fig. 8). 
Usually, a small feedback loop allows for sol- 
ving the problem of the scenario adaptation 
and verification successfully. However, if the 
problem is not solved within the small feed- 
back in a certain (small) number of steps, a 
large loop of feedback should be involved that 
tries to find another scenario Q),, which will 


be relevant to the current situation Sit. 

Thus, the control of the joint activity of 
a heterogeneous ensemble of unmanned ve- 
hicles using ISCS is a complex four-level 
process. 

At the operational level, the main task is 
to select a suitable operation plan and to build 
appropriate mission plans based on the current 
spatial configuration V(En,t). The operation 


scheme Sch(Op) should be adjusted to the 
structure Str(En) and shape Shp(En) of the 


ensemble En. Using the operational frame- 
work, the scheme Sch(Op) must be decom- 
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posed into a set of scenarios Q,, for all roles 
» assigned to the UV u, € Sir(En). 

At the mission level, each UV executes 
chosen scenarios concerning the roles assigned 
to it. The ISCS converts the scenario to the 
corresponding sequence of _ pentads 
(1,, Pos (u, )F (ust, ),Q;t;) taking into account 
the set of the given constraints. The ISCS recei- 
ves observations captured by sensors, trans- 
forms them into the event streams, and checks 
the spatial configuration V(En,t). In the case 


r,ER 


of the changes detected in the spatial configure- 
tion, if the mission plan is affected by these 
changes, the search for a new mission plan 
begins, otherwise, the coordination level proce- 
dure is called. 

At the coordination level, the sets of 
waypoints and timepoints of joint vehicle acti- 
vity should be agreed upon under the selected 
UV scenarios, then the admissibility of each 
waypoint/timepoint in the pentads should be 
evaluated to avoid dangerous areas and lack of 
schools of fish. 


Spatial 
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At the low control level, UVs receive a 
sequence of pentads at the input, generates the 
appropriate changes of the certain parameters’ 
values /f,,(t),...f,(t)¢(t), and transforms 
them in the sequence of low-level control ac- 
tions transferred to the UV’s actuators. 

The prototype of the ISCS including the 
implementation of both operational and coordi- 
nation frameworks has been developed based 
on the onboard microcontroller STM32F429 
(Cortex M4 processor 180 MHz, internal RAM 
256 KB, and flash memory 2 MB, external me- 
mory module QSPI Flash N25Q512). To deve- 
lop the ISCS, C++ programming language and 
GNU tools for embedded ARM processors 
have been used. 

Fig. 8 shows the place of the ISCS in 
the general structure of on-board equipment 
intended for the control of UVs’ joint activity 
within the heterogeneous ensemble during 
the fishing operation. 


> model 
! 
Observed 
parameters ‘Auitonanious trajectory 
———_________________» navigation UV propulsion 
scenarios controller system 


Safety areas deviation 
Motion 
Intelligent dusion | | | 
scenario-based =| 0 ttt 1 
control system 


positions obstacles 


Image recognition 


system 


Fig. 8. The general structure of the UV onboard system based on the ISCS 
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Conclusions 

The paper presents the idea of smart 
fishing operations based on the joint activity 
of the heterogeneous ensemble of intelligent 
unmanned vehicles of various types. This idea 
is embodied in the proposed “smart fishing” 
operation based on both the model of the joint 
fishing operation and the model of the hetero- 
geneous ensemble of unmanned vehicles re- 
garding its composition and structure. The 
operational framework presented in the paper 
includes missions, roles, plan prototypes, and 
activity scenarios, which are based on the dis- 
cretized spatial model. The coordination 
framework is also presented in the paper to 
solve the coordination task based on the distri- 
buted common board model and coordination 
primitives. The paper describes a prototype of 
the intelligent scenario-based system deve- 
loped for the control of unmanned vehicles in- 
cluding the implementation of both operatio- 
nal and coordination frameworks. It allows the 
UVs to perform smart fishing operations joint- 
ly in a large heterogeneous group. 

The proposed approach to smart fishing 
using intelligent technologies makes it pos- 
sible to detach fishermen from the fishing acti- 
vities dangerous to their life and health, to re- 
duce significantly poaching and illegal fishing, 
to increase the overall efficiency of fishing 
operations, and even to save the marine 
ecosystem. 
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